Method for generating images of multi-views

ABSTRACT

The present invention provides a method for generating images of multi-views. The method includes obtaining a 2D original image of an article and background figures of multi-views; calculating the background image range and the main body image range of the 2D original image of the article; cutting the main body image out; generating a depth model according to an equation; cutting the depth model according to the main body image range of the cut 2D image of the article; shifting every pixel in the main body image of the 2D original image of the article according to the cut depth model to obtain shifted main body images of multi-views; and synthesizing the shifted main body images of multi-views and the background figures of multi-views to obtain the final images of multi-views for 3D image reconstruction.

FIELD OF THE INVENTION

The present invention relates to three dimensional images, and moreparticularly to a method for generating images of multi-views.

BACKGROUND OF THE INVENTION

Three dimensional auto-stereoscopic image display technology has becomethe current trend in the image display technology field. Generally,three dimensional image display device can be classified as a glassestype and a bare-eye type. No matter what types of the three dimensionalimage display devices are used, the three dimensional image displaydevices render the user to see only a left eye image through the lefteye and only a right eye image through the right eye.

In the glasses type 3D image display devices, the most common onesinvolve the usage of polarization glasses, anaglyph glasses, shutterglasses, pulfrich glasses and a head mounted display. In the bare-eyetype 3D image display devices, the most common ones are a holographictype, a volumetric type, and a multiplexed 2-D type. The multiplexed 2-Dtype can be classified as a spatial-multiplexed type, a time-multiplexedtype and a tracking-based type.

If the bare-eye type 3D image display device, by which the user can seeauto-stereoscopic images without wearing any special devices, isdesigned based on the binocular parallax, the specific bare-eye type 3Dimage display device is referred to as a parallax based bare-eye type 3Dimage display device. This 3D image display device disposes a barrierfor example a parallax shield or a grating in front of light sourcearrays of a display for example a liquid crystal display, and the lightsource arrays transmit the left eye images and the right eye imagessequentially to render the left eye and the right eye of the user toreceive the left eye image and the right eye image independently throughthe parallax shield. Moreover, the technology which enables the user tosee auto-stereoscopic images without wearing any special devices alsoincludes integral photography auto-stereoscopic display technology. Themain technical theorem involves that holo-colored auto-stereoscopicimages are recorded through a fly's-eye lens, where the fly's-eye lensis a micro convex lens array arranged like a fly's eye, i.e. a pluralityof hemispheric small lenses arranged on a plane, to capture or displayimages.

However, no matter what types of devices are used to present the threedimensional images, the means used to generate different information forthe left eye and the right eye are similar. Generally, the signal sourcehas to provide different images for the left eye and the right eyerespectively. That is to say, two cameras are needed to capture the lefteye image and the right eye image. Therefore, a auto-stereoscopic camerawith two lenses is utilized to capture the left eye image and the righteye image in the conventional technique, and the left eye image and theright eye image are then output to the three dimensional display deviceto form a auto-stereoscopic image. However, the auto-stereoscopic camerawith two lenses can not be available easily to a common user becausethey are expensive. The user is unlikely to buy a three dimensionaldisplay device and enjoys no fun of the three dimensional images becausethe three dimensional image contents are not available. It is thereforehard to popularize the three dimensional display device for example athree dimensional display. Accordingly, there is still a need for atechnical solution which can solve the aforementioned problems of lackof the three dimensional image contents.

SUMMARY OF THE INVENTION

To solve the aforementioned problems of lack of the three dimensionalimage contents, the present invention provides a method for generatingimages of multi-views.

In one aspect, the present invention provides a method for generatingimages of multi-views, comprising obtaining a 2D original image of anarticle and background figures of multi-views, wherein the 2D originalimage of the article includes a main body image and a background image;calculating the range of the background image and the range of the mainbody image of the 2D original image of the article by a processing unit;cutting the main body image of the 2D original image of the article outto generate a cut 2D image of the article; generating a depth modelaccording to an equation; cutting the depth model according to the rangeof the main body image of the cut 2D image of the article to generate adepth model with a main body image outline; shifting every pixel in themain body image of the 2D original image of the article respectivelyaccording to the depth model with the main body image outline to obtainshifted main body images of multi-views; and synthesizing the shiftedmain body images of multi-views and the background figures ofmulti-views to obtain final images of multi-views for 3D imagereconstruction.

One advantage of the present invention is that the present invention canmake images of multi-views, for example the left eye image and the righteye image, for 3D image reconstruction by only utilizing a 2D camera incooperation with the system and the method without an expensive threedimensional camera.

Another advantage of the present invention is that the present inventioncan render the three dimensional image contents to be easy to beavailable. The acceptance of the three dimensional display device isimproved and the development of the three dimensional display device isaccelerated.

Still another advantage of the present invention is that the threedimensional images formed by the method and the system may contain thedepth of field in both the background and the main body image and willeliminate the montage issue.

These and other advantages will become apparent from the followingdescription of preferred embodiments taken together with theaccompanying drawings and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The present invention may be understood by some preferred embodimentsand detailed descriptions in the specification and the attached drawingsbelow. The identical reference numbers in the drawings refer to the samecomponents in the present invention. However, it should be appreciatedthat all the preferred embodiments of the invention are only forillustrating but not for limiting the scope of the Claims and wherein:

FIG. 1 illustrates a flow chart of a method for generating images ofmulti-views for 3D image reconstruction in accordance with oneembodiment of the present invention;

FIG. 2 illustrates a flow chart of the method for generating images ofmulti-views for 3D image reconstruction in accordance with oneembodiment of the present invention;

FIG. 3 illustrates a flow chart of the method for generating images ofmulti-views for 3D image reconstruction in accordance with oneembodiment of the present invention;

FIG. 4 illustrates a flow chart of a method for generating images ofmulti-views for 3D image reconstruction in accordance with anotherembodiment of the present invention;

FIG. 5 illustrates a flow chart of the method for generating images ofmulti-views for 3D image reconstruction in accordance with anotherembodiment of the present invention;

FIG. 6 illustrates a flow chart of the method for generating images ofmulti-views for 3D image reconstruction in accordance with anotherembodiment of the present invention;

FIG. 7 illustrates an example of a 2D original image of the articlehaving a pure color background in accordance with one embodiment of thepresent invention;

FIG. 8 illustrates a diagram showing the cutting procedure of the mainbody image in accordance with one embodiment of the present invention;

FIG. 9 illustrates a diagram showing the cutting procedure of the depthmodel in accordance with one embodiment of the present invention;

FIG. 10 illustrates a diagram showing the shifting procedure of thepixels in the main body image in accordance with one embodiment of thepresent invention;

FIG. 11 illustrates a diagram showing the synthesis procedure of themain body images and the background figures of different views inaccordance with one embodiment of the present invention;

FIG. 12 illustrates a diagram showing the depth model and therelationship between the brightness and the horizontal position thereofin accordance with one embodiment of the present invention;

FIG. 13 illustrates a diagram showing the depth model and therelationship between the brightness and the horizontal position thereofin accordance with another embodiment of the present invention;

FIG. 14 illustrates a block diagram of a system for generating images ofmulti-views for 3D image reconstruction in accordance with anotherembodiment of the present invention; and

FIG. 15 illustrates a block diagram of an example of a computer systemcooperating with the method and the system of the present invention inaccordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The invention will now be described with the preferred embodiments andaspects and these descriptions interpret structure and procedures of theinvention only for illustrating but not for limiting the Claims of theinvention. Therefore, except the preferred embodiments in thespecification, the present invention may also be widely used in otherembodiments.

The present invention provides a method, a system and a computerreadable medium for generating images of multi-views, for example a lefteye image and a right eye image, for 3D image reconstruction. Themulti-view images refer to the images viewed from two or more differentsites. The method of the present invention mainly utilizes thebackground image detecting technique in cooperation with the depthmodel, the depth image based rendering (DIBR) algorithm and the imagesynthesis technique to generate multi-view images, for example the lefteye image and the right eye image or the multi-view images, for 3D imagereconstruction by using a traditional 2D camera, so as to present thethree dimensional images by using various kinds of different threedimensional display devices.

As shown in FIG. 1, in one embodiment of the present invention, themethod for generating the left eye image and the right eye image for 3Dimage reconstruction includes obtaining a 2D original image of anarticle having a pure color background, a left eye background figure anda right eye background figure in step 101. The article may refer topeople, items or anything. Then, the 2D original image of the articlessuch as people or items is stored into a storage medium in step 101. Inone embodiment of the present invention, the pure color background maybe a single color background, such as a blue background, a greenbackground or backgrounds with any other colors which are not similar tothe color of the articles. The blue background may include a mazarinebackground. In one embodiment of the present invention, as shown inFIGS. 2 and 7, step 101 includes a step of taking a picture of thearticles in front of a pure color curtain with a 2D camera in step 1011.The pure color curtain may include a blue curtain, a green curtain orcurtains with any other colors which are not similar to the color of thearticle such as people or items. Subsequently, as shown in FIG. 1, thebackground image range and the main body image range of the 2D originalimage of the article are calculated by a processing unit in step 102.The main body image is the image of the article while the backgroundimage is the image of the pure color background. In one embodiment ofthe present invention, each color value, for example the red value (R),the green value (G) and the blue value (B), of every pixel in the 2Doriginal image of the article such as people or items is calculated bythe processing unit. If the value generated by subtracting the greenvalue (G) of one specific pixel in the 2D original image of the articlefrom the blue value (B) of the specific pixel is larger than 20 and thegreen value (G) is larger than the red value (R), the specific pixel isdetermined as the background image range. If not, the specific pixel isdetermined as the main body image range.

Then, as shown in FIGS. 1 and 8, the main body image of the 2D originalimage of the article is cut out to generate the cut 2D image of thearticle such as people or items in step 103. Subsequently, as shown inFIG. 1, a depth model is generated according to an equation in step 104.The center of the depth model is brighter than the two peripheral areasof the depth model. The brightness value of the center is 255 while thebrightness value of the two peripheral areas is 0. The brightness valuesof the areas are gradually decayed from the center to the two peripheralareas. In one embodiment, the equation includes a linear equation asshown in FIG. 12. In another embodiment, the equation includes anonlinear equation, for example Gamma curve as shown FIG. 13. In stillanother embodiment, the equation includes a polynomial equation. In yetanother embodiment, the equation is as follows: the brightness value ofthe left half portion of the depth model=(horizontalposition/(width/2))^0.45×255, the brightness value of the right halfportion of the depth model=((width−horizontalposition)/(width/2))^0.45×255. Then, as shown in FIGS. 1 and 9, thedepth model is cut according to the main body image range of the cut 2Dimage of the article such as people or items to generate a merged depthmodel with the main body image outline in step 105.

Subsequently, as shown in FIGS. 1 and 10, every pixel in the main bodyimage of the 2D original image of the article is shifted with the depthimage based rendering (DIBR) algorithm according to the depth model withthe main body image outline to obtain the left eye image of the mainbody image and the right eye image of the main body image in step 106.As shown in FIG. 3, step 106 may include step 1061 and step 1062. In oneembodiment, every pixel in the main body image of the 2D original imageof the article is shifted leftwards according to the gray levels of thedepth model with the main body image outline to obtain the right eyeimage of the main body image in step 1061. In another embodiment, everypixel in the main body image of the 2D original image of the article isshifted rightwards according to the gray levels of the depth model withthe main body image outline to obtain the left eye image of the mainbody image in step 1062. If the gray level of one specific pixel in thedepth model with the main body image outline is larger, the shift levelof the pixel in the 2D original image of the article corresponding tothe position of the specific pixel in the depth model will be larger.The gray level of the center of the depth model is the largest, and thegray levels of the two peripheral areas of the depth model are thesmallest. Therefore, if one specific position is closer to the center ofthe main body image, the shift level of the specific position is larger.If one specific position is closer to the two peripheral areas of themain body image, the shift level of the specific position is smaller.Then, as shown in FIGS. 1 and 11, the left eye image of the main bodyimage and the right eye image of the main body image are synthesizedrespectively with the left eye background figure and the right eyebackground figure to obtain final left eye image and final right eyeimage for 3D image reconstruction in step 107. In one embodiment of thepresent invention, the left eye background figure and the right eyebackground figure can be downloaded from the internet or be obtained bytaking a picture of a desired background, for example a scene, from twodifferent angles of view with a 2D camera. The final left eye image andthe final right eye image for 3D image reconstruction may be output tovarious kinds of three dimensional display devices, for example theglasses type three dimensional display device, the bare-eye type threedimensional display device or multi-view three dimensional displaydevice, to display the final left eye image and the final right eyeimage as a three dimensional image by utilizing different imagereconstructing methods.

As shown in FIGS. 4 and 5, in another embodiment of the presentinvention, the method of the present invention includes step 201 to step205 and step 2011 which are similar to step 101 to step 105 and step1011 in the aforementioned embodiment and therefore the redundantdescriptions are omitted. As shown in FIG. 4, every pixel in the mainbody image of the 2D original image of the article is shifted with aplurality of arithmetically increased shift levels respectively by usingthe depth image based rendering (DIBR) algorithm according to the depthmodel with the main body image outline to obtain the main body images ofmulti-views in step 206. As shown in FIG. 6, step 206 includes step 2061and step 2062. In one embodiment, a maximum shift level and a minimumshift level of every pixel in the main body image of the 2D originalimage of the article are determined according to the gray levels of thedepth model with the main body image outline in step 2061. Then,arithmetically increased intermediate shift levels are calculated withan arithmetic progression between the maximum shift level and theminimum shift level of every pixel in step 2062. Subsequently, as shownin FIG. 4, the main body images of multi-views are synthesizedrespectively with background figures of multi-views to obtain finalimages of multi-views for 3D image reconstruction in step 207. In oneembodiment of the present invention, the background figures ofmulti-views can be downloaded from the internet or be obtained by takinga picture of a desired background, for example a scene, from multipledifferent angles of view with a 2D camera. The final images ofmulti-views for 3D image reconstruction may be output to various kindsof three dimensional display devices, for example the glasses type threedimensional display device, the bare-eye type three dimensional displaydevice or multi-view three dimensional display device, to utilizedifferent image reconstructing methods to display the final images ofmulti-views as a three dimensional image.

As shown in FIG. 14, in still another embodiment of the presentinvention, the present invention provides a system for generating imagesof multi-views for 3D image reconstruction. The system includes aprocessing unit 301, an image range calculating module 302, a depthmodel generating module 303, an image cutting module 304, a pixelshifting module 305 and an image synthesizing module 306. The imagerange calculating module 302, the depth model generating module 303, theimage cutting module 304, the pixel shifting module 305 and the imagesynthesizing module 306 are coupled to the processing unit 301respectively. In one embodiment, the system further includes a shiftlevel calculating module 307 coupled to the processing unit 301. Theimage range calculating module 302 is utilized to calculate a backgroundimage range and a main body image range of a 2D original image of thearticle. The depth model generating module 303 is utilized to generate adepth model according to an equation. The equation includes a linearequation and a nonlinear equation. Preferably, the nonlinear equationincludes a Gamma curve. In one embodiment, the equation includes apolynomial equation. The image cutting module 304 is employed to cut the2D original image of the article or the depth model to generate a cut 2Dimage of the article or a depth model with the main body image outline.The pixel shifting module 305 is introduced to shift every pixel in themain body image of the 2D original image of the article according to thegray levels of the depth model with the main body image outline toobtain the shifted main body images of multi-views. The imagesynthesizing module 306 is utilized to synthesize the shifted main bodyimages of two or more views and the background figures of two or moreviews to obtain final images of two or more views for 3D imagereconstruction. The shift level calculating module 307 is utilized tocalculate two or more pairs of arithmetic shift levels according to thegray levels of the depth model with the main body image outline in orderfor the pixel shifting module 306 to shift every pixel in the 2Doriginal image of the article.

The method and the system for generating images of multi-views for 3Dimage reconstruction of the present invention can be performed incooperation with a computer system as follows. The computer system isshown for illustrating but not for limiting the present invention. Asshown in FIG. 15, the computer system includes a processing unit 401, amemory 402, a display 403, an input device 404, a network interface 405,a storage device 406 and a data transmission interface 407. The memory402, the display 403, the input device 404, the network interface 405,the storage device 406 and the data transmission interface 407 arecoupled to the processing unit 401 respectively. The input device 404may include a keyboard, a keypad or a mouse. The storage device 406 mayinclude a hard disk drive or a floppy drive. The network interface 405may include a wired network interface or a wireless network interface.The data transmission interface 407 may include a wired datatransmission interface and a wireless data transmission interface. Thewired data transmission interface may include universal serial bus (USB)and IEEE1394. The wireless data transmission interface may includeBLUETOOTH and infrared (IR).

The present invention may also be performed in other alternative formsbesides the aforementioned embodiments. In an alternative embodiment,the present invention provides a computer readable medium containingcomputer program instructions, which when executed by a computer, causethe computer to perform any of the aforementioned methods. It should beappreciated that the computer readable medium may be floppy drive, harddisk drive or any other mediums used to long-termly store computerexecutable instructions.

Accordingly, the method, the system and the computer readable medium forgenerating images of multi-views for 3D image reconstruction provided bythe present invention can generate multi-view images, for example theleft eye image and the right eye image or multi-view images, for 3Dimage reconstruction by only utilizing a 2D camera in cooperation withthe system and the method without an expensive three dimensional camera.Therefore, the three dimensional image contents can be easily obtained.The acceptance of the three dimensional display device can be improvedand the development of the three dimensional display device can beaccelerated. The three dimensional images formed from the multi-viewimages generated by the present method and the system can contain thedepth of field in both the background and the main body image and theissue of montage will not occurs.

The foregoing description is a preferred embodiment of the presentinvention. It should be appreciated that this embodiment is describedfor purposes of illustration only, not for limiting, and that numerousalterations and modifications may be practiced by those skilled in theart without departing from the spirit and scope of the invention. It isintended that all such modifications and alterations are includedinsofar as they come within the scope of the invention as claimed or theequivalents thereof.

What is claimed is:
 1. A method for generating images of multi-views for3D image reconstruction, comprising: obtaining a 2D original image of anarticle and background figures of multi-views, wherein said 2D originalimage of said article includes a main body image and a background image;calculating the range of said background image and the range of saidmain body image of said 2D original image of said article by aprocessing unit; cutting said main body image of said 2D original imageof said article out to generate a cut 2D image of said article;generating a depth model according to an equation; cutting said depthmodel according to said range of said main body image of said cut 2Dimage of said article to generate a depth model with a main body imageoutline; shifting every pixel in said main body image of said 2Doriginal image of said article respectively according to said depthmodel with said main body image outline to obtain shifted main bodyimages of multi-views; and synthesizing said shifted main body images ofmulti-views and said background figures of multi-views to obtain finalimages of multi-views for 3D image reconstruction.
 2. The method ofclaim 1, wherein said step of obtaining said 2D original image of saidarticle comprises a step of taking a picture of said article in front ofa pure color curtain with a 2D camera.
 3. The method of claim 1, whereinsaid every pixel in said main body image of said 2D original image ofsaid article is shifted with a plurality of arithmetically increasedshift levels respectively.
 4. The method of claim 1, wherein said stepof shifting every pixel in said main body image of said 2D originalimage of said article respectively according to said depth model withsaid main body image outline comprises: determining a maximum shiftlevel and a minimum shift level of said every pixel in said main bodyimage of said 2D original image of said article according to gray levelsof said depth model with said main body image outline; and calculating aplurality of arithmetically increased intermediate shift levels with anarithmetic progression between said maximum shift level and said minimumshift level of said every pixel.
 5. The method of claim 1, wherein saidequation comprises a linear equation or a nonlinear equation.
 6. Themethod of claim 5, wherein said nonlinear equation comprises a Gammacurve.
 7. The method of claim 1, wherein said equation comprises apolynomial equation.
 8. The method of claim 1, wherein said backgroundimage comprises a pure color background, wherein said pure colorbackground comprises a blue background or a green background.